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Management Improvements Needed on 
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Environment Project 

Highlights of GAO-02-545, a report to the Subcommittee on Treasury and General Government, Senate 
Committee on Appropriations, and the Subcommittee on Treasury, Postal Service, and General 



Why GAO Did This Study 

The U.S. Customs Service is in 
the early stages of a multiyear, 
multibillion-dollar project: the 
Automated Commercial 
Environment (ACE), a new 
import processing system that is 
to support effective and efficient 
movement of goods into the 
United States. By congressional 
mandate, Customs' spending 
plans for ACE must meet certain 
conditions, including being 
reviewed by GAO. In this study, 
GAO addresses whether 
Customs' latest plan satisfies 
congressional conditions and is 
consistent with open GAO 
recommendations, and it 
identifies opportunities for 
strengthening project 
management. 



What GAO Recommends 

To increase the chances of 
delivering needed system 
capabilities on time and within 
budget, GAO is making 
recommendations to the 
commissioner aimed at improving 
Customs' management of ACE, 
including strengthening system 
alignment with Customs' 
enterprise architecture, cost 
estimating, human capital 
capacity, software process 
maturity, and sequencing of 
incremental releases. Customs 
concurred with GAO's 
recommendations and described 
specific actions that it is taking to 
respond to each. 



What GAO Found 

Customs' February 2002 ACE spending plan is the second in a series of 
legislatively required plans. This plan covers certain project management 
tasks as well as the definition, design, and development of the first ACE 
increment. GAO found that the plan meets the legislative conditions that 
Congress imposed on Customs and is consistent with GAO's open 
recommendations. Nevertheless, investment in ACE is a high-risk 
endeavor for several reasons: 

• The system's size, performance parameters, and organizational impact 
make it technically and managerially complex. For example, the 
estimated cost for acquisition alone is about $1.5 billion, the system is 
to operate around-the-clock, and it is to drive fundamental business 
process reform not only within Customs, but also within numerous 
other federal agencies and countless private sector importers. 

• Customs fell far short of key commitments made in its first spending 
plan because it severely underestimated costs. This track record casts 
doubt on Customs' ability to meet future commitments. 

• Despite progress, Customs still lacks important acquisition 
management controls. For example, it needs to update its enterprise 
architecture (its agencywide modernization blueprint) to support 
system design and development, and it needs to advance its acquisition 
office's human capital and software management processes. 

• Customs has recently decided to compress its time frame for delivering 
the system from 5 to 4 years. This exacerbates the level of project risk 
by introducing more overlap among incremental system releases. 

Because of the system's national importance, Customs is taking a 
schedule-driven approach to acquiring ACE. However, without the 
management capacity to effectively acquire such a large and complex 
system, particularly in light of Customs' performance to date and the 
accelerated acquisition and deployment schedule, this approach could 
backfire. Full system capabilities may take longer and cost more to 
acquire, deploy, and make operational, because the system delivered 
under the accelerated schedule could require considerable rework. 

ACE Is High Risk for Several Reasons 
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United States General Accounting Office 
Washington, D.C. 20548 



May 13, 2002 

The Honorable Byron L. Dorgan 
Chairman 

The Honorable Ben Nighthorse Campbell 
Ranking Minority Member 

Subcommittee on Treasury and General Government 
Committee on Appropriations 
United States Senate 

The Honorable Ernest J. Istook, Jr. 
Chairman 

The Honorable Steny H. Hoyer 
Ranking Minority Member 
Subcommittee on Treasury, Postal Service, 

and General Government 
Committee on Appropriations 
House of Representatives 

Pursuant to the U.S. Customs Service's fiscal year 2002 appropriation, 1 
Customs submitted to Congress in February 2002 its second expenditure 
plan, requesting $206.9 million for its Automated Commercial Environment 
(ACE) project. ACE is intended to be Customs' new import processing 
system and the first project under the Customs Modernization Program. As 
required by the act, we reviewed the expenditure plan. Our objectives were 
to (1) determine whether the second plan satisfied the legislative 
conditions specified in the act, (2) determine whether the plan is consistent 
with our open ACE recommendations, and (3) provide observations about 
the second plan and Customs' management of ACE. 

On March 8 and 11, 2002, we briefed your offices on the results of this 
review. This report officially transmits to you the results of our work and 
the recommendations we made to the commissioner of Customs. The full 
briefing, including our scope and methodology, is reprinted as appendix I. 

In brief, we determined that Customs' expenditure plan satisfies the 
legislative conditions specified in the appropriations act. The plan provides 
for (1) meeting the Office of Management and Budget's (OMB) capital 
planning and investment control review requirements; (2) complying with 



1 Public Law 107-67, dated Nov. 12, 2001. 
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Customs' enterprise architecture; 2 and (3) complying with federal 
acquisition rules, requirements, guidelines, and systems acquisition 
management practices. Further, it was reviewed and approved by Customs' 
Investment Review Board, 3 the Department of the Treasury, and OMB. 

Regarding the second objective, activities described in the plan are 
consistent with our open recommendations for Customs to (1) justify and 
make investment decisions incrementally; (2) strengthen ACE software 
acquisition management; (3) immediately transfer responsibility and 
accountability for the International Trade Data System 4 (ITDS) pilot to the 
ACE modernization program manager; (4) include any plans for further 
investing in the ITDS pilot, including cost, benefit, and risk justification, in 
the next ACE expenditure plan; and (5) clarify the roles and responsibilities 
of the ACE modernization independent verification and validation 5 (IV&V) 
agent to ensure that independence is not compromised. 

Finally, we made the following observations: 

• ACE is technically and managerially complex and challenging. Planned 
ACE functional and performance parameters are demanding, and the 
system is estimated to cost at least $1.5 billion just to put in place, not 
including operation and maintenance. Also, ACE aims to fundamentally 
change Customs' and many other organizations' business processes 
through the introduction of new system capabilities. Further, the system 
is to be available around-the-clock to provide critical information and 
support important commercial and enforcement systems. 

• Customs did not meet key commitments made in its first ACE 
expenditure plan because of a significant underestimation of funding 
requirements (actual requirements were about 90 percent higher than 



2 An enterprise architecture is an institutional blueprint for guiding and constraining 
investments in business process change and systems. 

3 Customs' Investment Review Board makes information technology funding decisions on 
the basis of comparisons and trade-offs among competing project proposals. 

4 ITDS is a component of the ACE project that is to support governmentwide collection, use, 
and dissemination of trade data. 

5 The purpose of IV&V is to provide an independent review of system processes and 
products. The use of IV&V is a recognized best practice for large and complex systems 
development and acquisition projects. To be effective, IV&V should be performed by an 
entity that is independent of the processes and products that are being reviewed. 
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estimated). This history casts uncertainty on Customs' ability to reliably 
estimate costs and meet future commitments. 

• Despite progress, Customs does not yet have important ACE 
management controls in place in the following four areas: 

1. Enterprise architecture: Customs' enterprise architecture has not yet 
been updated and extended to support ACE engineering tasks; 
consequently Customs risks having an enterprise architecture that is 
not sufficiently complete to support ACE design activities. As a result, 
near-term system design and development decisions could later 
necessitate system rework to align ACE with the agency's operating 
vision, embodied in the updated enterprise architecture. If such rework 
is necessary, promised system capabilities are unlikely to be delivered 
on time and within budget. 

2. Human capital: ACE is being managed by the Customs Modernization 
Office (CMO). However, this office does not have the people in place to 
perform critical system acquisition functions. Moreover, the CMO does 
not have an effective strategy for meeting its human capital needs. This 
also increases the risk that promised system capabilities will not be 
delivered on time and within budget. 

3. Software acquisition management: Customs has yet to establish 
software acquisition process controls that are recognized as best 
management practices. 6 In particular, Customs has not begun to 
establish process controls for determining whether acquired software 
products and services satisfy contract requirements before acceptance, 
nor to establish related controls for effective and efficient transfer of 
acquired software products to the support organization responsible for 
maintenance. These control weaknesses further reduce the level of 
assurance that ACE capabilities will be delivered on time and within 
budget. 



6 These best practices are described in the Software Acquisition Capability Maturity Model 
(SA-CMM), developed by the Software Engineering Institute (SEI). In this model, SEI has 
provided criteria for characterizing an organization's software development and acquisition 
processes according to five levels of maturity with level 2 providing the minimum level of 
acceptable effectiveness. The Capability Maturity Model is a service mark of Carnegie 
Mellon University and CMM is registered in the U.S. Patent and Trademark Office. 
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4. Cost estimation: Customs has not validated its ACE expenditure plan 
cost estimates, and its estimates of management reserve costs are not 
justified. As a result, its plan does not provide Congress with credible 
cost information needed for overseeing the project. 

• Customs has compressed its ACE acquisition plans from 5 to 4 years. 
This compression increases the degree of overlap of program 
increments, which in turn further increases the risk that ACE 
capabilities will not be delivered on time and within budget. 



RGCOmmGndSltionS for ^° i m P rove Customs' ACE modernization management, we recommend 

. . a . . that the Customs Service commissioner direct the chief information officer, 

JLXCCUllVC f\Cll(JiL ^ e designated modernization executive, to take the following actions: 



• Before building each ACE release (i.e., beginning detailed design and 
development), certify to Customs' House and Senate appropriations 
subcommittees that the enterprise architecture has been sufficiently 
extended to provide the requisite enterprise design content and has 
been updated to ensure consistency and integration across business 
areas. 



• Immediately develop and implement a CMO human capital management 
strategy that provides both near- and long-term solutions to the CMO's 
human capital capacity limitations, including defining the office's skill 
and capability needs in terms that will allow Customs to attract qualified 
individuals and that will provide sufficient rewards and training, linked 
to performance, to promote their retention. 

• Develop and implement process controls for the SEI SA-CMM level 2 
key process areas and the level 3 acquisition risk management key 
process area. 

• By September 30, 2002, conduct and report to Customs' House and 
Senate appropriations subcommittees on the results of an internal 
assessment of the CMO's maturity against the SEI SA-CMM level 2 key 
process areas and the level 3 acquisition risk management key process 
area. 



• Develop and implement a rigorous and analytically verifiable cost 
estimating program that embodies the tenets of effective estimating as 
defined in SEI's institutional and project-specific estimating models. 
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• Limit future expenditure plan requests for management reserve funds to 
10 percent of the total funds requested for the program or adequately 
justify any management reserve requests in excess of 10 percent. 

• Address the risks associated with the accelerated ACE acquisition 
strategy including (1) immediately analyzing the risks associated with 
the degree of design, development, and testing concurrency across ACE 
increments that is inherent in Customs' 4-year, schedule-driven 
acquisition strategy; (2) reconsidering the merits of this accelerated 
strategy; and (3) within 90 days of the date of this briefing, reporting to 
Customs' House and Senate appropriations subcommittees on the 
agency's strategy for going forward and its plans for mitigating the 
inherent risks associated with this strategy 



In written comments on a draft of this report, the director, Office of 
Planning, U.S. Customs Service, concurred with our recommendations and 
described specific actions that are being taken to respond to each. The 
director's comments are reprinted in appendix II. 



We are sending copies of this report to the chairmen and ranking minority 
members of other Senate and House committees and subcommittees that 
have authorization and oversight responsibilities for the Customs Service. 
We are also sending copies to the secretary of the treasury, the 
commissioner of the Customs Service, and the director of OMB. 

Should you or your staff have any questions on matters discussed in this 
report, please contact me at (202) 512-3439. 1 can also be reached by e-mail 
at HiteR@gao.gov . Key contributors to this report were Mark Bird, Barbara 
Collier, Tamra Goldstein, Randolph Tekeley, Scott Pettis, and Aaron 
Thorne. 




Randolph C. Hite 

Director, Information Technology Architecture 
and Systems Issues 



Agency Comments 
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Subcommittee on Treasury and General Government 
Senate Committee on Appropriations 

and 

Subcommittee on Treasury, Postal Service, and General Government 
House Committee on Appropriations 

March 8 and 11, 2002 
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Briefing Overview 


. Introduction 




• Objectives, Scope, and Methodology 




• Results in Brief 




• Background 




• Results 




• Conclusions 




• Recommendations for Executive Action 




• Agency Comments 
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The U.S. Customs Service is acquiring a new trade processing system known as 
the Automated Commercial Environment (ACE)ZInternational Trade Data 
System (ITDS). 1 This project is intended to fundamentally transform not only 
Customs' internal import processing operations, but also the manner in which 
it and numerous other federal agencies interact with each other and with the 
trade community. According to Customs, the project will, for example, 

• enable more efficient import processing; 

. minimize the gap between revenue that should be collected and revenue that 
is actually collected; 

• increase the flow of legitimate commerce; and 

. provide the trade community a single interface with 104 federal agencies that 
have trade-related missions. 



1 Hereafter referred to as ACE. 
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Customs' FY 2001 and FY 2002 appropriations acts 2 provided $130 million and 
$300 million, respectively, for ACE. The acts specified that these funds not be 
obligated until Customs submits to the Congress for approval ACE expenditure 
plans that 

• meet the Office of Management and Budget's (OMB) capital planning and 
investment control review requirements; 

• comply with the Customs Service's enterprise architecture; 

• comply with federal acquisition rules, requirements, guidelines, and systems 
acquisition management practices; 

• are reviewed and approved by the Customs Investment Review Board, the 
Department of the Treasury, and OMB; and 

• are reviewed by GAO. 



' Public Laws 106-554 and 107-67, respectively. 
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On February 1, 2002, Customs submitted its second ACE expenditure plan to 
its House and Senate appropriations subcommittees, seeking release of $206.9 
million from the ACE appropriations. 
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Objectives 

As agreed, our objectives were to 

. determine whether the second ACE expenditure plan satisfies the legislative 
conditions, 

• determine whether the plan is consistent with our open ACE 
recommendations, and 

. provide observations about the plan and Customs management of ACE. 
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Scope and Methodology 

To accomplish our objectives, we 

• analyzed the plan against the legislative conditions, to determine if the 
conditions were met; 

• analyzed the plan to determine consistency of actions taken by Customs in 
response to our open recommendations; 

. interviewed ACE and ITDS program management, support contractor, and 
modernization contractor officials to understand the scope and content of 
plans, and to clarify information in supporting documentation; 

• analyzed ACE supporting documentation to ensure consistency with 
information in the expenditure plan and to determine progress made in 
implementing program management capabilities (e.g., acquisition 
management, enterprise architecture, human capital, and cost and schedule 
estimating); 



Page 7 of 60 



Page 12 



GAO-02-545 Customs Service Modernization 



Appendix I 

Customs' Second Automated Commercial 
Environment (ACE) Expenditure Plan 



i 

jGAO 

^^^Accountablllty * Integrity * Reliability QbjeCtjveS, SCOpe and MetItOdolOgy (COnt.) 

• determined progress made against commitments in first expenditure plan; 

• coordinated with the Treasury inspector general to avoid duplication of 
effort in reviewing ACE; 

• did not evaluate the ACE Investment Strategy and Cost Benefit Analysis, 
which we received on February 13, 2002; and 

• as agreed, did not independently validate Customs' provided information, 
including cost estimates and project content and status information. 

We performed our work from December 2001 through March 2002 in 
accordance with generally accepted government auditing standards. 



Page 8 of 60 



Page 13 



GAO-02-545 Customs Service Modernization 



Appendix I 

Customs' Second Automated Commercial 
Environment (ACE) Expenditure Plan 




Results in Brief: Objective 1 



Customs' expenditure plan satisfies the legislative conditions. 



Legislative conditions Satisfies Does not satisfy 



1. Meets OMB capital planning and investment control review 




requirements, including OMB Circular A-ll, part 3. 3 




2. Complies with the Customs Service's enterprise architecture. 




3. Complies with federal acquisition rules, requirements, guidelines, and 




systems acquisition management practices. 




4. Is reviewed and approved by the Customs Investment Review Board, 


s 


Treasury, and OMB. 




5. Is reviewed by GAO. 





3 Customs provided us with a draft of the expenditure plan before it was approved by Treasury and OMB. We told Customs 
that the version provided did not satisfy OMB Circular A-ll, part 3, requirements. Customs amended the plan before 
submitting it to the House and Senate Appropriations Committees. 
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Customs' expenditure plan is consistent with open GAO 




recommendations. 




GAO open recommendations Consistent 


Inconsistent 


Complete In progress 




1. Justify and make investment decisions incrementally. S 




2. Strengthen ACE software acquisition management. S 




3. Immediately transfer responsibility and accountability S 




for the ITDS pilot to the ACE modernization program 




manager. 




4. Include any plans for further investing in the ITDS pilot, S 




including cost, benefit, and risk justification, in the next 




ACE expenditure plan. 




5. Clarify the roles and responsibilities of the ACE S 




modernization independent verification and validation 




(IV&V) agents to ensure that independence is not 




compromised. 
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Results in Brief: Objective 3 



Other observations: Customs faces considerable risk in acquiring 
and implementing ACE. 



1 . ACE is technically and managerially complex 
and challenging. 



Adds risk 
+ 



2. Customs did not meet key commitments made 
in its first ACE expenditure plan. 

3. Despite progress, Customs does not yet have 
important ACE management controls in place. 



4. Customs has accelerated its ACE acquisition 
plans. 



-► Adds risk 
+ 



Adds risk 
+ 

-► Adds risk 





CUMULATIVE 
RISK IS HIGH 
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Background 



ACE Overview 

ACE is intended to support Customs' efforts to protect our nation's borders, 
enforce trade laws, and facilitate the movement of legitimate trade. For 
example, ACE is to 

• support all Customs functions associated with trade compliance (such as 
defining, capturing, and managing regulations for import and export 
compliance; establishing and managing accounts for trade participants; and 
verifying credentials and managing compliance levels); 

• support other functions such as data analysis for trade law enforcement and 
border security purposes; 

• support the trade information collection, processing, and dissemination 
needs of 104 other government agencies; and 

• provide the trade community a single interface to the federal government. 
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According to Customs, ACE will permit, for example, more efficient import and 
export operations; increased legitimate cargo flow; reduced paper burden; and 
more efficient and effective account management, including revenue billing 
and collection operations. 

We have testified that the need for Customs to modernize its trade processing 
capability is undeniable because, among other things, Customs' existing import 
processing system was paper-intensive, error-prone, transaction-based, and out 
of step with the just-in-time inventory practices of the trade community. 4 



4 U.S. General Accounting Office, U.S. Customs Service: Observations on Selected Operations and Program Issues, 
GAO7T-GGD/AIMD-00-150 (Washington, D.C.: Apr. 20, 2000). 
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Background (cont.) 



ACE Acquisition Strategy 

Customs is using an integration contractor, supported by subcontractors, to 
develop and deploy ACE. The contractor and its subcontractors are 
collectively called the e-Customs Partnership or eCP. 

Initial modernization contract activities include establishing a management 
foundation (e.g., people, processes, tools) for Customs' overall modernization 
program. ACE is to be the first modernized system under this program. 

Customs' current plans call for release of ACE capabilities in four increments 
implemented over 4 years. Each increment is to contain discrete functionality, 
while building on the functionality in the previous increments. 
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ACE Management Structure 




Executive Steering Committee 



I 



Customs 



Assistant Commissioner/ Office of Information Technology 
Chief Information Officer 
Modernization Executive 




I 

Customs 
Modernization Office 
(CMO) 



Business units 



CIO Directors 




| Support contractors : 
MITRE 
Robbins-Gioia 



Government task 
coordinators 



Government task 
coordinators 




eCP 



IBM Global Services 



Lockheed Martin 




Computer Sciences 






Corporation 



I 



KPMG Consulting 



Sandler & Travis 
Trade Advisory Services 



Other subcontractors: 



Booz -Allen Hamilton Over 40 small businesses 
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Chronology of ACE Expenditure Plans and Funding 

On March 5 and 6, 2001, Customs' House and Senate appropriation 
subcommittees, respectively, released to Customs $5 million in stopgap 
funding to sustain program activities until Customs submitted its first 
expenditure plan. 

On March 26, 2001, Customs submitted its first ACE expenditure plan to 
Congress for $45 million. 

On April 23, 2001, we briefed your offices on the results of our review of the 
first ACE expenditure plan. In summary, we reported that 

• the plan satisfied the legislative conditions; 

• the plan was consistent with our February 1999 recommendations that 
Customs 

■ justify and make ACE investment decisions incrementally, and 

■ strengthen ACE software acquisition management; 
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Customs was investing in an ITDS pilot project in parallel with, and separate 
from, ITDS efforts under the ACE modernization program; and 

Customs had defined conflicting roles and responsibilities for its 
independent verification and validation (IV&V) contractor. 



Accordingly, we made the following recommendations, which Customs agreed 
to implement: 

. Immediately transfer responsibility and accountability for the ITDS pilot to 
the ACE modernization program manager. 

• Include any plans for further investing in the ITDS pilot, including cost, 
benefit, and risk justification, in the next ACE expenditure plan. 

• Clarify the roles and responsibilities of the ACE modernization IV&V agent to 
ensure that independence is not compromised. 
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On April 24 and 25, 2001, Customs' House and Senate appropriation 
subcommittees, respectively, released to Customs an additional $45 million in 
response to Customs' first ACE expenditure plan. 

On April 27, 2001, Customs awarded a 5-year, indefinite-delivery indefinite- 
quantity contract, with options to extend the contract to not more than 15 
years, to International Business Machines (IBM) Global Services. 

On February 1, 2002, Customs submitted its second ACE expenditure plan, 
seeking release of $206.9 million. 

The following two figures illustrate the recent history of ACE appropriations, 
expenditure plans, funding, and major contract events. 
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Background (cont.) 



ACE funding releases shown by fiscal year appropriations 



FY 2001 appropriations = $130 million 



FY 2002 appropriations = $300 million 



March 2001 
$5 million 
funding 
release 



April 2001 
$45 million 
funding 
release 




Second expenditure 
plan for $206.9 million 




Next expenditure 
plan (planned for 
May 2002) 
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Background (cont.) 



Illustrated chronology of ACE expenditure plans and funding 



Contract 
awarded 



$5M 
release 



$45M 
release 



First 
plan 
submitted 



Contractor 
authorized to 
proceed 



Initial three 
task orders 
signed 



Second 
plan 
submitted 



Next plan 



12/00 



3/01 



6/01 



9/01 



12/01 



3/02 



6/02 
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Customs' second ACE expenditure plan requests funding for the 
following: 

Areas of expenditure Amount (millions) 

Funds to cover first expenditure plan cost variance 5 $22.4 

National Customs Automation Program (NCAP) operations & maintenance $3.1 

Program management and support $23.5 

SAP 6 infrastructure $7.3 

ACE increment 1, release 1.0 design, development, test, and deployment $104.0 

ACE increment 2, detailed requirements and planning $11.2 

Customs Modernization Office (CMO) management reserve $35.5 

Total $206.9 



5 The second expenditure plan requests funding equivalent to the difference between the negotiated price for the task 
orders ($47.4 million) and the funding released as a result of its first expenditure plan ($25 million). 

6 These funds will be used to acquire hardware and supporting infrastructure for Customs' SAP Implementation Project, a 
joint project of the Office of Finance and the Office of Information Technology. SAP AG is a vendor that makes a software 
package called R/3, which is to be used by ACE to account for payments, collections, and refunds. Additionally, R/3 is to 
replace Customs' existing core budget, accounting, procurement, and property accountability systems. 
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Background (cont.) 


Customs attributes its first expenditure plan variance to underestimates 
or omissions in the following areas: 


Labor: $12.9 million (58 percent of the total variance) 


Variance (millions) 


Labor rates and skill mix 

Tasks (e.g., change management, Stakeholder Relations Council) 
Other (e.g., performance-based contracting) 


$6.8 
$4.5 
$1.6 


Other direct cost: $9.9 million (42 percent of the total variance) 


Task order planning 

Facilities and support infrastructure 

Other (e.g., performance-based contracting, travel costs) 


$2.0 
$6.7 
$1.2 


Total 


$22.8 
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Results: Objective 1 



Customs' expenditure plan satisfies the legislative conditions. 



Legislative conditions 



Satisfies? Results of our analysis 



1. Plan meets OMB capital 
planning and investment control 
review requirements, including 
OMB Circular A-l 1, part 3. 



S OMB circular A-ll, part 3 sets requirements for budget 
justification and reporting requirements for major 
acquisitions. To respond to A-ll requirements, 
agencies are to cover key topics such as: 

• compliance with federal security and privacy 
regulations; 

• compliance with Government Paperwork 
Elimination Act requirements; 

• compliance with strategic planning requirements 
that demonstrate how the project will use a 
performance-based measurement system; and 

• cost and schedule information. 

The expenditure plan and supporting documentation 
address each of these key topics. 
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Results: Objective 1 (cont.) 




Legislative conditions Satisfies? 


Results of our analysis 


2. Plan complies with Customs' S 
enterprise architecture. 


The expenditure plan and supporting documentation 
provide for the development of an ACE system 
architecture that is to comply with Customs' enterprise 
architecture. 


3. Plan complies with federal S 
acquisition rules, requirements, 
guidelines, and systems 
acquisition management 
practices. 


The expenditure plan and supporting documents 
include, for example, steps for implementing the 
Software Engineering Institute's (SEI) Software 
Acquisition Capability Maturity Model (SA-CMM) 7 key 
process areas. If implemented effectively, these 
processes should enable Customs to meet important 
federal acquisition rules, requirements, guidelines, and 
systems acquisition management practices. 


Capability Maturity Model SM is a service mark of Carnegie Mellon University, and CMM is registered in the U.S. Patent and 
Trademark Office. The SA-CMM identifies key process areas that are necessary to effectively manage software-intensive 
system acquisition. Level 2 is the second level of the SA-CMM's five-level scale and means that an organization has the 
software acquisition rigor and discipline to repeat project successes. 
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Results: Objective 1 (cont.) 


^ ^Accountability * Integrity * Reliability 




Legislative conditions Satisfies? 


Results of our analysis 


4. Plan is reviewed and approved S 
by Customs Investment Review 
Board (IRB), Treasury, and OMB. 


Customs IRB approved the expenditure plan in two 
parts on September 26, 2001, and October 16, 2001. 
Treasury approved the plan on December 20, 2001. 
OMB approved the plan on January 31, 2002. 


5. Plan is reviewed by GAO. S 


Our review was completed on March 1, 2002. 
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Customs' expenditure plan is consistent with GAO open 
recommendations. 




GAO recommendations 



Consistent? Actions planned or taken 



Complete 



In 

progress 



1. Justify and make investment 
decisions incrementally, and for 
each increment: 

• Use disciplined investment 
processes; 



• Prepare realistic and supportable 
benefit expectations; 

• Require a favorable return on 
investment and compliance with 
Customs architecture before 
making an investment decision; 
and 

• Validate actual costs and benefits 
once an increment is piloted. 



S Customs expenditure plan and February 12, 2002, ACE 
Investment Strategy and Cost Benefit Analysis 8 describe 
an incremental approach to investing in ACE that 

• Entails compliance with Customs investment 
management process, which we determined meets 
OMB's policy for management of federal information 
resources; 

• Identifies costs and benefits for the first ACE 
increment; 

• Identifies a positive return on investment and 
compliance with Customs' enterprise architecture, as 
required by Customs' investment management 
process; and 

• Plans for validation of actual costs and benefits as 
determined in post-implementation reviews. 



* We did not evaluate the ACE Investment Strategy and Cost Benefit Analysis, which we received on February 13, 2002. 
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GAO recommendations 



Consistent? Actions planned or taken 



Complete 



2. Strengthen ACE software 
acquisition management by 

• establishing an effective 
process improvement program 
and correcting any weaknesses 
in ACE software acquisition 
processes identified in this 
report thereby bringing ACE 
processes to at least SEI CMM 
level 2; and 

• requiring at least CMM level 2 
processes of all ACE software 
contractors 



S Customs' expenditure plan includes funds to 

continue software acquisition process improvement 
efforts, including 

• establishment of a software acquisition process 
improvement program, correction of software 
acquisition processes, and demonstration of 
progress toward bringing ACE processes to at 
least SEI SA-CMM level 2. 



• Customs instituted a policy requiring CMM level 
2 processes of all software contractors, and is 
requiring that its prime modernization contractor 
maintain level 3 processes. 
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Results: Objective 2 (cont.) 
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GAO recommendations 


Consistent? 

Complete In 

progress 


Actions planned or taken 


3. Immediately transfer 
responsibility and accountability 
for ITDS pilot to the ACE 
modernization program manager. 




Effective May 2, 2001, Customs transferred all ITDS 
program management responsibilities to the 
Customs Modernization Office, which is headed by 
the ACE modernization program manager. 


4. Include any plans for further 
investing in the ITDS pilot 
(including cost, benefit, and risk 
justification) in the next ACE 
expenditure plan. 




Customs suspended the ITDS pilot on September 11, 
2001, to devote all available resources to 
maintaining its "Level One Alert" status at border 
crossings. Customs does not plan to restart the ITDS 
pilot. 


5. Clarify the roles and 
responsibilities of the ACE 
modernization IV&V agent to 
ensure that independence is not 
compromised. 




Customs has clarified the ACE IV&V agent's role by 
eliminating the agent's program support 
responsibilities in areas where it has IV&V 
responsibility. 
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Results: Objective 3 



Other observations: Customs faces considerable risk in acquiring 
and implementing ACE. 



1 . ACE is technically and managerially complex 
and challenging. 



-► Adds Risk 
+ 



2. Customs did not meet key commitments made 
in its first ACE expenditure plan. 

3. Despite progress, Customs does not yet have 
important ACE management controls in place. 



4. Customs has accelerated its ACE acquisition 
plans. 



-► Adds Risk 
+ 

-► Adds Risk 
+ 




Adds Risk 




CUMULATIVE 
RISK IS HIGH 
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Observation 1: ACE is technically and managerially complex and 

challenging. 

Planned ACE functional and performance parameters are demanding. For 
example, ACE is to 

• be deployed and used at about 300 ports of entry. 

• be available 24 hours a day, 365 days a year. 9 

. provide functionality currently provided by about 15 other systems, 
including the following: 

■ Automated Commercial System (which tracks, controls, and 
processes all goods imported into the United States); 

■ Automated Export System (which allows electronic transmissions 
of key export data from exporters, exporters' agents, and outbound 
carriers); 



A specific system availability requirement has not yet been defined. 
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■ Treasury Enforcement Communications System (which is to 
support inspection personnel at borders and ports of entry along 
with other mission areas such as investigations, interdiction, and 
intelligence analysis); and 

■ Seized Currency and Asset Tracking System (which tracks all 
activity associated with seizures from the initial enforcement 
interest in the property until its final disposition); 

process about 29 million transactions annually; 10 

account for over $20 billion in revenue annually; and 

require the integration of several large commercial off-the-shelf (COTS) 
software products using customized software (i.e., glue code). 11 



According to Customs, it processed about 29 million import transactions in 1998. 

11 Customs did not yet have software size and complexity measures. Further, while the suite of COTS products to be used 
has yet to be defined, Customs plans indicate that COTS packages will likely include customer relationship management 
software, electronic document management and workflow software, workflow product and collaborative groupware 
software, relational database management system (RDMS) software, and online analytical processing and rules software. 
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ACE is a large undertaking, as measured by the level of development resources 
needed. In particular, the cost of developing the entire system is estimated to 
be between $1.5 and $1.6 billion, and the number of professional staff needed 
to develop the system is estimated to be about 500 to 700. 

ACE is also intended to support and satisfy the needs of 104 other government 
agencies with international trade-related missions as well as the needs of 
thousands of importers. Accordingly, acquiring a system that meets this diverse 
set of needs will require an enormous level of stakeholder participation, 
cooperation, and interaction. 

ACE is intended to fundamentally change not only the way that Customs 
operates internally but also the way that importers and other government 
agencies operate. The ability of these organizations to accept and absorb such 
operational change is not guaranteed; their uncertainty has been recognized as 
a reason that COTS-based modernization efforts have not consistently met 
expectations. 
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Observation 2: Customs did not meet key commitments made in its 

first ACE expenditure plan. 

One indication of future performance, and thus risk, is past performance. 
Customs' past performance in acquiring ACE fell short of the commitments it 
made in the first expenditure plan. 

In the first plan, Customs requested $25 million for three ACE tasks: 
(1) program management, (2) enterprise engineering, and (3) requirements 
development. Customs reports that it met commitments for task 3, but it did 
not meet all of the commitments for tasks 1 and 2. Specifically, because the 
cost to complete the three tasks collectively increased $22.8 million over what 
Customs requested in the first plan, Customs funded the first two tasks for only 
6 months, rather than 18 months as specified in its expenditure plan. (See table 
on next page.) 
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Results: Objective 3 (cont.) 





Task 


Summary of commitment in first plan 


Actual result 


Impact on 
second plan 



1. Program Establish a program office to manage the 
management modernization program for an 18-month period. 

2. Enterprise Establish and maintain the systems engineering 
engineering capability for implementing ACE for an 18 month 

period. This work includes, for example, updating 
and maintaining Customs' Enterprise Architecture, 
Enterprise Life Cycle Model, and Technical 
Reference Model. 

3. ACE Develop high-level ACE system requirements, and 
requirements detailed increment 1 requirements, as well as ACE 
and planning project plan outlining the transition from ACS to 

ACE. This work was to be done within 6 months. 



Work was 
performed for 6 
months. 

Work was 
performed for 6 
months. 



ACE system 
requirements, 
including the 
detailed 

requirements for 
ACE increment 1, 
along with 
project plan, 
were delivered 
within 6 months. 



$22.4 million to 
fund the 
remaining 12 
months on the 
first two task 
orders. 



None. 
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According to Customs, the $25 million request in the first expenditure plan was 
based on a 1999 estimate. Customs officials acknowledged that derivation of 
this estimate lacked sufficient rigor because it was not based on detailed work 
statements. 

Customs' inability to meet its initial commitments casts uncertainty on its 
ability to meet future commitments and thus introduces an element of risk to 
the ACE project. 
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Observation 3: Despite progress, Customs does not yet have key 

management controls for ACE. 

a. Extension of the enterprise architecture to support ACE 
engineering tasks is not complete. 

Enterprise architectures are corporate blueprints to guide and constrain 
operational and technological change, particularly investments in large system 
development or acquisition efforts, such as ACE. To effectively guide and 
constrain these efforts, enterprise architectures should be updated to ensure 
that they reflect changes in the strategic business direction of the enterprise. 
Further, they should be updated in a way that ensures consistency and 
integration across the full scope of the enterprise's business areas; for 
Customs, these areas are commercial, enforcement, and administration. Also, 
these architectures should be sufficiently extended to recognize system- 
specific business requirements upon which later design decisions will be 
based. Such extensions are both normal and a necessary part of 
methodologically sound system design and development. To minimize risk, 
these architectural activities should be completed before system design and 
development begins. 
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Customs recognizes the need to ensure that its ACE system architecture is 
further defined, and it has instituted processes for providing such assurance. 
For example, it is having eCP extend the ACE system architecture to include 
logical data models and data flows and expand the standards for Web services. 
However, Customs has yet to commit to work on its enterprise architecture 
that eCP officials say is needed to ensure that Customs' enterprise architecture 
is internally consistent, sufficiently scoped, and adequately extended. 

Instead, Customs' plans to provide "just enough" enterprise architecture "just- 
in-time" to guide and constrain the design of future ACE increments. According 
to eCP officials, parallel definition of the enterprise and system architectures is 
not typically done. The reason it is not typically done is because it risks not 
having sufficient enterprise context available to effectively design the system. 

We believe that the reason for Customs' parallel approach is the priority being 
given to meeting the ACE schedule. In our view, such a schedule-driven 
approach risks producing an enterprise architecture extension that is not 
sufficiently complete to support ACE design activities; this, in turn, could 
result in costly system rework and a failure to deliver system capabilities on 
time and within budget. 
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b. Customs does not have an effective strategy for meeting its 
modernization human capital needs. 

Effective human capital management is fundamental to any organization (or 
organizational component), such as the CMO. To be effective, information 
technology (IT) human capital management strategies should provide for 

• identifying knowledge, skills, and experience requirements; 

• continuously assessing existing capabilities; and 

• developing effective strategies and implementing plans to fill any gaps, 
including developing compensation packages to attract and retain qualified 
individuals. 

Since the CMO's establishment in June 2000, its approach to meeting its human 
capital needs has been to identify a set of CMO positions, develop position 
descriptions for each, advertise these positions, and select among applicants. 

Using this approach, the CMO currently does not have the human capital 
capacity it needs to effectively manage ACE. (See table on next page.) To 
compensate, Customs' approach has been to rely on support contractors. 
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Results: Objective 3 (cont.) 



Status of CMO positions: 



Position title 


Level 


Filled? 


Number of months vacant 


Executive Director 


SES 


Yes 




Deputy Director 


GS-15 


No 


2 


Requirements Executive 


GS-15 


Yes 




Requirements Executive 


GS-15 


Yes 




Executive Support 


GS-14 


Yes 




Executive Support 


GS-7 


Yes 




Requirements Management Lead 


GS-14 


No 


7 


Business Management Lead 


GS-14 


No 


5 


Communications Lead 


GS-14 


No 


10 


Program Control Lead 


GS-14 


No 





Systems Engineering Lead 


GS-13/14 


No 


9 


Program Analyst 


GS-9/13 


No 





Program Analyst 


GS-9/13 


No 


5 
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In our view, the CMO's lack of adequate human capital can be traced to not 
having defined and implemented an effective human capital strategy. Such a 
strategy would, at a minimum, completely and correctly define the CMO's 
human capital needs (both the proper skill levels and the commensurate 
compensation packages needed to attract and retain them) and provide 
Customs' full commitment (resources and management attention) for pursuing 
short-term and long-term actions to meet them. Unless Customs addresses its 
human capital weaknesses, it increases the risk of not delivering ACE 
capabilities on time and within budget. 
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c Software acquisition management processes are incomplete. 

Effective IT management processes, such as processes for acquiring software, 
are essential to software-intensive system acquisitions. SEI, recognized for its 
expertise in software processes, has defined the tenets of effective software 
acquisition. The SA-CMM identifies, among other things, a number of key 
process areas that are necessary to effectively manage software-intensive 
system acquisition. 

As discussed earlier, Customs is addressing our recommendations aimed at 
correcting the ACE software acquisition management weaknesses that we 
identified in our February 1999 report, 12 and Customs officials told us that they 
plan to eventually assess their SA-CMM capabilities, although no time frame 
has been set. 



U.S General Accounting Office, Customs Service Modernization: Serious Management and Technical Weaknesses Must 
Be Corrected, GAO/AIMD-99-41, Washington, D.C., Feb. 26, 1999. 
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Using Customs' reported data on the status of its efforts to address these 
weaknesses, we determined that one of the SEI key process areas is generally 
complete (i.e., the vast majority of practices in a key process area are in place), 
five key process areas are works in progress (i.e., one or more practices in a 
key process area are in place), and two key process areas are not started (i.e., 
no practices in a key process area are in place). 
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Status at Customs on SEI key process areas: 



SEI SA-CMM key process area Generally Work in Not started 

complete progress 

Software acquisition planning: ensuring that reasonable planning ^ 
for the software acquisition is conducted and that all elements of 
the project are included. 

Solicitation: ensuring that award is made to the contractor most ^ 
capable of satisfying the specified requirements. 



Requirements development and management: establishing a 
common and unambiguous definition of software acquisition 
requirements understood by the acquisition team, system user, and 
the contractor. 

Project office management: managing the activities of the project 
office and supporting contractor(s) to ensure a timely, efficient, and 
effective software acquisition. 

Contract tracking and oversight: ensuring that the software 
activities under contract are being performed in accordance with 
contract requirements, and that products and services will satisfy 
contract requirements. 
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SEI SA-CMM key process area Generally Work in Not started 

complete progress 

Evaluation: determining that the acquired software products and ^ 
services satisfy contract requirements prior to acceptance and 
transition to support. 

Transition to support: providing for the transition of the software ^ 

products being acquired to the eventual software support 

organization. 

Acquisition risk management: 13 identifying risks as early as possible, ^ 
adjusting acquisition strategy to mitigate those risks, and developing 
and implementing a risk management process as an integral part of 
the acquisition process. 



Acquisition risk management is an SEI SA-CMM level 3 key process area. 
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As shown in the table above, Customs has yet to complete many key processes 
and, in particular, the evaluation and transition to support key process areas 
have not been started. The purpose of evaluation is to determine that the 
acquired software products and services satisfy contract requirements before 
acceptance. Evaluation should include developing and managing evaluation 
requirements in conjunction with developing software technical requirements. 
Transition to support is to provide for the effective and efficient "hand off of 
the acquired software products to the support organization responsible for 
software maintenance and should include designating a group that is 
responsible for coordinating transition and support activities. 

Customs officials agree that its software acquisition processes are incomplete 
and told us that Customs delayed addressing the evaluation and transition to 
support key process areas because resources (i.e., staff) were needed for 
higher priority activities. In the absence of an evaluation capability, Customs 
said that they are using a support contractor to perform evaluation activities. 
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Because Customs has not yet established effective software acquisition 
processes, it does not have adequate assurance that ACE capabilities will be 
delivered on time and within budget. 
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d. Expenditure plan cost estimates have not been validated by 
Customs. 

Producing reliable cost estimates for inclusion in ACE expenditure plans is 
essential. Without such estimates, Customs cannot provide congressional 
committees with the information needed to exercise important ACE oversight. 
Ensuring that cost estimates are reliable requires, at a minimum, that Customs 
satisfy itself that rigorous, analytically verifiable methods are used to derive 
the estimates. To this end, SEI has defined models for (1) establishing an 
institutional estimating capability and (2) validating the reliability of a project- 
specific estimate. 14 

As discussed earlier, Customs' cost estimates for the initial ACE tasks in its 
first expenditure plan were not reliable because Customs did not use a 
rigorous approach in deriving them. 



Software Engineering Institute, Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities 
of Software Organizations fCMU/SEI-95-SR-005) and A Manager's Checklist for Validating Software Cost and Schedule 
Estimates fCMU/SEI-95-SR-004). 
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For its second expenditure plan, Customs did not use any method to derive 
contract-related costs. Instead, it simply passed along estimates developed by 
the ACE contractor, except for Customs' internal (CMO) costs and the 
management reserve. According to Customs, it plans to use an independent 
contractor to validate cost estimates after it submits future expenditure plans 
and to use this information in negotiating task orders. 

For the management reserve of $35.5 million requested in the second 
expenditure plan, Customs used a contractor to derive this estimate. This 
estimate represents about 21 percent of the total funds requested (excluding 
the reserve) in the expenditure plan. We found that this reserve estimate 
lacked analytical justification and relied extensively on the unsubstantiated 
assumptions and subjective judgments of a few individuals. In fact, we could 
have used the same methodology used by the contractor, made our own 
equally valid assumptions and judgments, and produced estimates either 
considerably higher or lower. 
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Moreover, available information from studies and comparable system 
undertakings, such as the Internal Revenue Service's Business Systems 
Modernization (BSM), do not support this level of management reserve. For 
example, the BSM management reserve is 10 percent, and a study covering 
about 500 Defense contracts showed a mean management reserve of about 4 
percent. 

We conclude that the management reserve cost estimate appears overstated, 
and the reliability of the other estimates in the second expenditure plan is 
uncertain. Without reliable estimates, Customs increases the risk that it will 
repeat its first expenditure plan performance, and that it will not meet 
expenditure plan commitments. 
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Observation 4: Customs has accelerated its ACE acquisition plans. 

When a system is incrementally acquired, it is important to carefully manage 
the extent to which design, development, testing, and implementation of the 
increments overlap (i.e., occur concurrently). The goal is to ensure that a given 
increment has been sufficiently matured (so that it is free of material defects 
and performs as intended) before later, dependent increments are built. 

While some degree of increment concurrency is appropriate, extensive overlap 
of increments' detailed design, development, and testing activities is generally 
not desirable because it increases the risk of overall system rework. Reasons 
for this include, carrying over unresolved defects into later increments; 
maintaining control over a proliferating number of system versions; stretching 
critical resources (such as people, development and testing laboratories); and 
introducing incremental change into the organization with such frequency that 
it exhausts the organization's capacity to absorb the change. 
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As previously stated, Customs has committed to implementing our prior 
recommendation to incrementally acquire and invest in ACE. Until recently, 
Customs' incremental strategy provided for acquiring and deploying ACE in 
four major increments spanning 5 years, with the first three of the four 
increments consisting of two subincrements or releases. Even under this 5-year 
strategy, Customs' plans provided for some overlap among increments. For 
example, design activities in one increment were generally occurring 
concurrently with development activities in later increments. (See figure on 
next page.) 

In January 2002, Customs decided to accelerate its plans for acquiring and 
deploying ACE, electing to deliver ACE's four increments in 4 years rather than 
5 because, according to the Commissioner of Customs, ACE is important to our 
country's security and the future of trade facilitation. While Customs has yet to 
develop a program schedule detailing the extent to which this compressed 
schedule will increase concurrency among system increment design, 
development, and testing, Customs and its integration contractor acknowledge 
that increased concurrency among increments will occur. 
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Results: Objective 3 (cont.) 



ACE 5-year implementation plan: 




Requirements & planning 
Design 
Develop 
Deploy 



y 
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The contractor has also acknowledged that this increased concurrency will 
exacerbate preexisting risks, such as 

• the possibility that an increment's requirements will not be fully understood 
before design and development is to begin; 

• the possibility that the delivery of COTS and hardware products will not be 
timely; and 

• the possibility that changes will be required to Customs' legacy systems with 
which ACE must interface. 



The contractor has also stated that the compressed schedule introduced new 
risks, such as 

• funding requirements will be so high that budget approval is unlikely; 

• staffing requirements will be so high that physical housing of all personnel 
will be difficult; and 

• the trade community will not have sufficient lead time to make changes 
necessary to support ACE. 
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Combined with the other risks that we cited earlier (i.e., unresolved defects 
carrying over to later increments, version control, overtaxing core resources, 
and capacity for change absorption), the chances of not delivering ACE 
capabilities on time and within budget are increased appreciably. 
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Customs' second ACE expenditure plan satisfies the legislative conditions 
and is consistent with our open recommendations. Nevertheless, ACE 
introduces a host of unavoidable risks, including the complexity of the 
work, the dependence of successful implementation on the interaction of 
a large and diverse set of stakeholders, and these stakeholders' ability to 
absorb the considerable operational changes to be introduced through 
ACE. 

One indicator of Customs' ability to successfully address these risks and 
deliver ACE capabilities on time and within budget is its past 
performance. Customs did not fulfill important commitments that it made 
in its first expenditure plan because the plan's estimated costs for certain 
tasks were not reliable. Because of this track record, the risks in moving 
forward on ACE are amplified. 
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Another indicator of Customs' ability to successfully acquire and 
implement ACE is the effectiveness of the management controls it has in 
place to govern its system acquisition and implementation activities. For 
various reasons cited by Customs officials, including limited resources 
and schedule demands, Customs has yet to establish several important 
management controls, such as an updated enterprise architecture, 
adequate human capital capacity, complete software acquisition 
management processes, and reliable cost estimating. The absence of 
these controls further compounds the risks facing Customs on ACE. 

Notwithstanding these risks, Customs has recently decided to develop 
and deploy the system faster. This decision exacerbates an already risky 
undertaking by increasing the difficulty of the work without first 
providing for a commensurate increase in the capabilities needed to 
perform the work. 
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To address the risks discussed in this briefing, we recommend that the 
Customs Service Commissioner direct the Chief Information Officer, as the 
designated Modernization Executive, to take the following actions: 

• Before building each ACE release (i.e., beginning detailed design and 
development), certify to Customs' House and Senate appropriation 
subcommittees that the enterprise architecture has been sufficiently 
extended to provide the requisite enterprise design content and has been 
updated to ensure consistency and integration across business areas. 

• Immediately develop and implement a CMO human capital management 
strategy that provides both near-term and long-term solutions to the CMO's 
human capital capacity limitations, including defining the office's skill and 
capability needs in terms that will allow Customs to attract qualified 
individuals and that will provide sufficient rewards and training, linked to 
performance, that promote their retention. 
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Develop and implement process controls for the SEI SA-CMM level 2 key 
process areas and the level 3 acquisition risk management key process area. 

By September 30, 2002, conduct and report to Customs' House and Senate 
appropriation subcommittees on the results of an internal assessment of the 
CMO's maturity against the SEI SA-CMM level 2 key process areas, and the 
level 3 acquisition risk management key process area. 

Develop and implement a rigorous and analytically verifiable cost estimating 
program that embodies the tenets of effective estimating as defined in SEI's 
institutional and project-specific estimating models. 

Limit future expenditure plan requests for management reserve funds to 10 
percent of the total funds requested for the program or adequately justify any 
management reserve requests in excess of 10 percent. 
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Recommendations for Executive Action 



Address the risks associated with the accelerated ACE acquisition strategy, 
including (1) immediately analyzing the risks associated with the degree of 
design, development, and testing concurrency across ACE increments that is 
inherent in Customs' 4-year, schedule-driven acquisition strategy; (2) 
reconsidering the merits of this accelerated strategy; and (3) within 90 days 
of the date of this briefing, reporting to Customs' House and Senate 
appropriations subcommittees on the agency's strategy going forward and its 
plans for mitigating the inherent risks associated with this strategy. 
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In commenting on a draft of this briefing, the Customs Modernization Office 
Executive Director agreed with our findings, conclusions, and 
recommendations. 
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U.S. Customs Service 

Memorandum 

DATE: April 24, 2002 

FILE: AUD-1-OPMD 



MEMORANDUM FOR RANDOLPH C. HITE 

U.S. GENERAL ACCOUNTING OFFICE 

FROM: Director 

Office of Planning 

SUBJECT: Draft Audit Report on the United States Customs 

Service's Second Automated Commercial 
Environment (ACE) Expenditure Plan 

Thank you for providing us with a copy of your draft report entitled 
"Customs Service Modernization: Management Improvements Needed on 
High-Risk Automated Commercial Environment Project" and the 
opportunity to discuss the issues in this report. 

We agree with GAO's overall observations that the Customs 
Modernization program is large, complex and important, and thus 
represents considerable risk. We have taken, and will continue to take 
prudent steps to address the risks associated with the Modernization 
program. Key actions are highlighted below: 

• While we believe that the Customs Modernization Office (CMO) was 
appropriately staffed and structured for the early phases of the 
program, our early experiences also led to recognition that a new 
structure and additional resources were required to oversee 
application development and deployment. A new, expanded 
organizational structure has been approved doubling the size of the 
CMO. These positions are being filled now. 

• We have taken a number of steps to improve our cost estimating 
capability including use of an independent government cost estimator 

Tradition and implementation of software tools that allow for a reliable cost 

estimation process. These efforts will be reflected in future Expenditure 
* Plans. 
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Although the original cost estimate differed from the actual cost, this is 
not representative of Customs current cost estimating capabilities. The 
cost estimates from the first ACE expenditure plan pre-dated the 
contract award and did not include task order changes recommended 
by eCP and approved by the CMO. Once the scope changes were 
approved, Task Orders 1-3 have come in on time, on budget, and 
without modification. Customs believes that now that the task 
statements are being developed collaboratively with eCP, and with the 
cost estimation improvements that CMO has implemented, the issue of 
large cost variances will pass. 

• Customs and eCP recognize the risks associated with pursuing a 
4-year application rollout. We have identified the inherent risks of this 
4-year approach, analyzed the potential impacts, and briefed the 
Modernization Executive Steering Committee. The CMO and eCP are 
employing risk management procedures that conform to Systems 
Engineering Institute standards to mitigate risks associated with the 
entire Modernization program, including the first ACE release. Prior to 
receiving authorization to proceed in each of the subsequent 6 
releases, we commit to re-assessing the risk environment and 
developing mitigation strategies to meet cost and quality targets. At 
any point, if risks cannot be credibly mitigated Customs and its 
Integration Contractor will recommend schedule adjustments. 

• Customs has tasked eCP to update and extend the existing enterprise 
architecture in concert with ACE requirements, planning and 
engineering work to provide the requisite design content, and 
consistency and integration across the full scope of the Customs 
business areas. We will do this in an incremental and logical fashion 
to minimize risks of subsequent re-work. We believe we can 
accomplish this objective by completing the basic enterprise 
architecture frameworks prior to beginning detailed design for each 
increment. Thus, the enterprise architecture frameworks and relevant 
derivative artifacts pertinent to the specific ACE increment content will 
be sufficiently complete prior to beginning detailed design for each 
increment. 
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• The CMO is proceeding with implementation of software acquisition 
controls that comply with the Software Engineering Institute's Software 
Acquisition Capability Maturity Model and is 75 percent complete. The 
CMO continues to focus on the most critical plans, processes and 
procedures first across all KPAs as we proceed through the 
Modernization effort. 

The attachment to this memorandum details the specific actions that are 
being taken to respond to the recommendations. 

If you have any questions regarding these comments, please contact 
Ms. Michele Donahue at (202) 927-0957. 




Attachment 
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Attachment 

U.S. CUSTOMS SERVICE 
GAO Review of Second Automated Commercial Environment 
Expenditure Plan 



Recommendation 1 : Before building each ACE release (i.e., beginning 
detailed design and development), certify to Customs' House and Senate 
appropriation subcommittees that the enterprise architecture has been 
sufficiently extended to provide the requisite enterprise design content and has 
been updated to ensure consistency and integration across business areas. 

Response: 

Customs recognizes and supports the imperative to establish and incrementally 
enhance the enterprise architecture to effectively guide and constrain large 
system development or acquisition projects such as ACE. Customs has tasked 
eCP to update and extend the existing enterprise architecture in concert with 
ACE requirements, planning and engineering work in order to provide the 
requisite design content, and consistency and integration across the full scope of 
the Customs business areas. The tasking and schedules are specifically 
designed to ensure the necessary enterprise architecture framework is in place 
to adequately guide and constrain each release of the ACE system, while also 
reflecting any resulting changes in the strategic business direction of the 
enterprise. 

Customs agrees with the GAO recommendation to certify that extensions to the 
enterprise architecture provide sufficient context to effectively design the system 
prior to building each ACE release and reflect any necessary strategic business 
updates required to ensure consistency and integration across all business 
areas. eCP has recommended tailoring of Customs System Development Life 
Cycle (SDLC) to include a Preliminary Design Review (PDR) which will include 
an enterprise and ACE system architecture assessment component. This review 
occurs prior to beginning detailed design and development, and therefore is the 
proper point in the lifecycle to demonstrate the necessary architectural 
framework and content exists to proceed. The results of this review will provide 
the objective evidence necessary to provide certification of the enterprise 
architecture to the House and Senate appropriation subcommittees. 

Customs expects to demonstrate compliance with this recommendation prior to 
the start of Task Order 4 detailed design work in June 2002. 

Target Date: June 30, 2002 

Responsible Executive: Charles Armstrong 
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Recommendation 2: Immediately develop and implement a CMO human 
capital management strategy that provides both near-term and long-term 
solutions to the CMO's human capital capacity limitations, including defining the 
office's skill and capability needs in terms that will allow Customs to attract 
qualified individuals and that will provide sufficient rewards and training, linked to 
performance, that promote their retention. 

Response: 

Customs believes it did apply appropriate program management resources to the 
initial phases of the Modernization program. However, Customs concurs with 
GAO that additional resources are required to manage the expanding complexity 
of the project. To that end, the CMO has defined a new organizational structure 
for managing the developmental phases of the Modernization program that more 
than doubles the government positions from 1 1 to 23. Knowledge and skills 
required to perform each role have already been identified. Customs is 
proceeding to staff the new organizational structure now, and by May 1 , 2002, 
the CMO will have filled a number of positions with full-time, Customs- 
experienced individuals leading the efforts in executive support, program 
management, workforce transition, integration, ACE requirements development, 
and ACE Increments 1 and 2 implementation. New CMO members will be quickly 
oriented to the Modernization Program and provided necessary training as 
defined in the Modernization Training Plan for Fiscal Year 2002. Customs will 
continue an aggressive recruiting program both within and external to Customs to 
fully staff the CMO. Further, the CMO will work with appropriate other Customs 
Offices to develop a human resources strategy by September 30, 2002 that will 
focus on defining core competencies to support recruiting, retention and training 
efforts; tying performance and reward programs to Modernization goals; and 
developing a succession plan to enable Customs to effectively adapt to 
personnel changes. 

Target Date: September 30, 2002 
Responsible Executive: Charles Armstrong 
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Recommendation 3: Develop and implement process controls for the SEI 
SA-CMM level 2 key process areas and the level 3 acquisition risk management 
key process area. 

Response: 

The CMO's comprehensive SA-CMM process development effort is 75 percent 
complete and spans all of SEI's SA-CMM Level 2 Key Process Areas (KPAs)— 
including the Evaluation and Transition to Support KPAs— and to the Level 3 
KPA, Acquisition Risk Management (ARM). As originally noted in the CMO's 
briefing to GAO on February 26, 2001 , the CMO continues to focus on the most 
critical plans, processes and procedures first across ail KPAs as we proceed 
through the modernization effort. 

As the processes are completed, they are immediately released to the 
organization for use in an institutionalization effort. Process institutionalization 
will continue after the last plan, process and procedure is approved and will end 
when all SA-CMM process artifacts are in systematic use within the CMO. 

The CMO anticipates that the process development activities will culminate in an 
internal self-assessment that will be completed by September 2002, and the 
CMO will use the September internal self-assessment to baseline the process 
institutionalization activities as well. The capstone of the institutionalization effort 
will be a SA-CMM external pre-assessment at the end of December 2002. The 
results of the external pre-assessment will be used to schedule the full SEI 
assessment in 2003. 

Target Date: September 30, 2002 
Responsible Executive: Charles Armstrong 
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Recommendation 4: By September 30, 2002, conduct and report to 
Customs' House and Senate appropriation subcommittees on the results of an 
internal assessment of the CMO's maturity against the SEI SA-CMM level 2 key 
process areas, and the level 3 acquisition risk management key process area. 

Response: 

The CMO's current Process Improvement schedule supports the GAO's 
recommendation to conduct and report to Customs' House and Senate 
appropriation subcommittees on the results of an internal assessment of the 
CMO's maturity against SA-CMM Level 2 KPAs and the Level 3 acquisition risk 
management KPA. Customs will report the results of its internal self-assessment 
by September 30, 2002. 

Target Date: September 30, 2002 
Responsible Executive: Charles Armstrong 



Recommendation 5: Develop and implement a rigorous and analytically 
verifiable cost estimating program that embodies the tenets of effective 
estimating as defined in SEI's institutional and project-specific estimating models. 

Response: 

Customs agrees with the GAO recommendation that a rigorous and analytically 
verifiable cost-estimating program is essential to the success of ACE. Customs 
has taken measures that have proven successful in providing a solid foundation 
on which the program has effectively contracted for three large Task Orders with 
no cost over-runs. 

While there was a variance from the original cost estimates for Task Orders 1 
and 2, that delta was due to a two-year old estimate, and a significant change in 
scope. To remedy the situation, Customs employed an independent contractor, 
MCRI to develop Independent Government Cost Estimates (IGCEs) in line with 
SEI's estimating checklist. When the IGCEs were delivered and the subsequent 
contract negotiations took place, Customs was able to issue Task Orders 1-3 that 
have come in on-time, on-budget, and without modification. Using the IGCE 
method to provide independent validation of eCP estimates, Customs has 
recently contracted for two more Task Orders (4 and 5) that fell well within the 
acceptable range of the initial estimates from MCRI. 

Customs is always exploring ways to refine its cost estimating practices, and will 
implement improvements as they are identified. One such improvement that has 
already been implemented is the use of Crystal Ball® risk analysis software to 
calculate levels of certainty around different estimating values. This software tool 
was used in formulating the Expenditure Plan that is currently undergoing 
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Executive Branch review prior to submission to Congress. This software tool, 
employing elements of SEI's cost estimation checklist, will be used to formulate 
all subsequent expenditure plans, elevating the estimating rigor, providing a 
greater level of confidence in the estimates, and allowing for better planning for 
known program risks. 

Target Date: Customs believes it has taken actions responsive to this 
recommendation. 

Responsible Executive: Charles Armstrong 

Recommendation 6: Limit future expenditure plan requests for 
management reserve funds to 10 percent of the total funds requested for the 
program or adequately justify any management reserve requests in excess of 10 
percent. 

Response: 

The CMO understands GAO concerns regarding management reserve. To 
better define risk and uncertainty, CMO is using Crystal Ball®, a risk analysis tool. 
Consequently, Customs will have adequate justification for future expenditure 
plan requests for management reserve funds. 

Target Date: Customs believes it has taken actions responsive to this 
recommendation. 

Responsible Executive: Charles Armstrong 

Recommendation 7: Address the risks associated with the accelerated 
ACE acquisition strategy, including (1) immediately analyzing the risks 
associated with the degree of design, development, and testing concurrency 
across ACE increments that is inherent in Customs 4-year, schedule-driven 
acquisition strategy, (2) reconsidering the merits of this accelerated strategy, and 
(3) within 90 days of the date of this briefing, reporting to Customs House and 
Senate appropriations subcommittees on the Agency's strategy going forward 
and its plans for mitigating the inherent risks associated with this strategy. 

Response: 

Customs and its Integration Contractor have already identified the inherent risks 
of this 4-year approach, analyzed the potential impacts, and briefed the 
Modernization Executive Steering Committee. The CMO and eCP are employing 
risk management procedures that conform to Systems Engineering Institute 
standards to mitigate risks associated with the entire Modernization program, 
including the first ACE release. Prior to receiving authorization to proceed in 
each of the subsequent 6 releases, the risk environment will be re-assessed and 
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mitigation strategies will be developed to meet cost and quality targets. At any 
point, if risks cannot be credibly mitigated, Customs and its Integration Contractor 
will recommend schedule adjustments. As suggested by the GAO, Customs will 
also provide a report to the House and Senate Appropriation Committees by 
April 30, 2002, outlining the risks identified and the corresponding mitigation 
plans. Within this same time period, Customs will also reconsider and analyze 
the merits of the current ACE program plan and include the findings in this report. 

Target Date: April 30, 2002 

Responsible Executive: Charles Armstrong 
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The General Accounting Office, the investigative arm of Congress, exists to 
support Congress in meeting its constitutional responsibilities and to help improve 
the performance and accountability of the federal government for the American 
people. GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance to help 
Congress make informed oversight, policy, and funding decisions. GAO's 
commitment to good government is reflected in its core values of accountability, 
integrity, and reliability. 
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